Primary and auxiliary image capture devices for image processing and related methods

ABSTRACT

Disclosed herein are primary and auxiliary image capture devices for image processing and related methods. According to an aspect, a method may include using primary and auxiliary image capture devices to perform image processing. The method may include using the primary image capture device to capture a first image of a scene, the first image having a first quality characteristic. Further, the method may include using the auxiliary image capture device to capture a second image of the scene. The second image may have a second quality characteristic. The second quality characteristic may be of lower quality than the first quality characteristic. The method may also include adjusting at least one parameter of one of the captured images to create a plurality of adjusted images for one of approximating and matching the first quality characteristic. Further, the method may include utilizing the adjusted images for image processing.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 13/584,744, filed Aug. 13, 2012, which claims the benefit ofU.S. patent application Ser. No. 13/115,589, filed May 25, 2011; whichclaims the benefit of U.S. Provisional Patent Application Ser. No.61/427,278, filed Dec. 27, 2010, the contents of all of which are herebyincorporated by reference in their entireties.

TECHNICAL FIELD

The subject matter disclosed herein relates to image processing. Inparticular, the subject matter disclosed herein relates to primary andauxiliary capture devices for image processing and related methods.

BACKGROUND

Stereoscopic, or three-dimensional, imagery is based on the principle ofhuman vision. Two separate detectors detect the same object or objectsin a scene from slightly different positions and/or angles and projectthem onto two planes. The resulting images are transferred to aprocessor which combines them and gives the perception of the thirddimension, i.e. depth, to a scene.

Many techniques of viewing stereoscopic images have been developed andinclude the use of colored or polarizing filters to separate the twoimages, temporal selection by successive transmission of images using ashutter arrangement, or physical separation of the images in the viewerand projecting them separately to each eye. In addition, display deviceshave been developed recently that are well-suited for displayingstereoscopic images. For example, such display devices include digitalstill cameras, personal computers, digital picture frames, set-topboxes, high-definition televisions (HDTVs), and the like.

The use of digital image capture devices, such as digital still cameras,digital camcorders (or video cameras), and phones with built-in cameras,for use in capturing digital images has become widespread and popular.Because images captured using these devices are in a digital format, theimages can be easily distributed and edited. For example, the digitalimages can be easily distributed over networks, such as the Internet. Inaddition, the digital images can be edited by use of suitable softwareon the image capture device or a personal computer.

Digital images captured using conventional image capture devices aretwo-dimensional. It is desirable to provide methods and systems forusing conventional devices for generating three-dimensional images. Inaddition, it is desirable to provide systems for providing improvedtechniques for processing images for generating three-dimensional imagesor two-dimensional images.

SUMMARY

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter.

Disclosed herein are primary and auxiliary image capture devices forimage processing and related methods. According to an aspect, a methodmay include using primary and auxiliary image capture devices, eachincluding an image sensor and a lens, to perform image processing. Themethod may include using the primary image capture device to capture afirst image of a scene, the first image having a first qualitycharacteristic. Further, the method may include using the auxiliaryimage capture device to capture a second image of the scene. The secondimage may have a second quality characteristic. The second qualitycharacteristic may be of lower quality than the first qualitycharacteristic. The method may also include adjusting at least oneparameter of one of the captured images to create a plurality ofadjusted images for one of approximating and matching the first qualitycharacteristic. Further, the method may include utilizing the adjustedimages for image processing.

According to another aspect, a method may include using primary andauxiliary image capture devices to generate a still image withoutblurring. The method may include using the primary image capture deviceto capture a first image of a scene. Further, the method may includeusing the auxiliary image capture device to capture a plurality of otherimages of the scene. The method may also include determining a motionblur kernel based on the plurality of other images. Further, the methodmay include applying the motion blur kernel to remove blur from thefirst image of the scene.

According to another aspect, a system may include a computing deviceincluding a primary image capture device and an external deviceinterface. The system may also include an attachable, computing deviceaccessory including an auxiliary image capture device and a connectorconfigured for operable attachment to the external device interface. Theaccessory may be configured to communicate captured images to thecomputing device via the connector and external device interface.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary, as well as the following detailed description ofvarious embodiments, is better understood when read in conjunction withthe appended drawings. For the purposes of illustration, there is shownin the drawings exemplary embodiments; however, the presently disclosedsubject matter is not limited to the specific methods andinstrumentalities disclosed. In the drawings:

FIG. 1 is a block diagram of an exemplary image capture system includinga primary image capture device an auxiliary image capture device for usein capturing images of a scene and performing image processing accordingto embodiments of the presently disclosed subject matter,

FIG. 2 is a flow chart of an exemplary method for performing imageprocessing using the system shown in FIG. 1, alone or together with anyother suitable device, in accordance with embodiments of the presentdisclosure;

FIGS. 3A and 3B are perspective views of an example video cameraincluding a primary image capture device and an auxiliary image capturedevice according to embodiments of the present disclosure;

FIG. 4 is a perspective view of an example digital camera including aprimary image capture device and an auxiliary image capture deviceaccording to embodiments of the present disclosure;

FIG. 5 illustrates steps in an example method for creating astereoscopic image pair in accordance with embodiments of the presentdisclosure;

FIG. 6 is a flow chart of an exemplary method for 3D constructionaccording to embodiments of the present disclosure;

FIG. 7 is a flow chart of an exemplary method for frame matching inaccordance with embodiments of the present disclosure;

FIG. 8 is a flow chart of an exemplary method for image view matchingaccording to embodiments of the present disclosure;

FIG. 9 is a flow chart of an exemplary method for color and luminancematching according to embodiments of the present disclosure;

FIGS. 10A and 10B depict a flow chart of an exemplary method forgenerating a high-dynamic range image using primary and auxiliary imagecapture devices according to embodiments of the present disclosure;

FIG. 11 is a flow chart of an exemplary method for removing motion blurusing a system having primary and auxiliary image capture devicesaccording to embodiments of the present disclosure;

FIG. 12 is a flow chart of an exemplary method for video stabilizationin a system having primary and auxiliary image capture devices accordingto embodiments of the present disclosure;

FIGS. 13A and 13B are perspective views of an example mobile telephonewith an auxiliary image capture device in a position to face a user andanother position to face a scene together with a primary image capturedevice according to embodiments of the present disclosure;

FIG. 13C illustrates a cross-sectional view of another example mobiletelephone according to embodiments of the present disclosure;

FIGS. 14A, 14B, and 14C are different views of another example mobiletelephone with an auxiliary image capture device in a position to face auser and another position to face a scene together with a primary imagecapture device according to embodiments of the present disclosure;

FIGS. 15A, 15B, 15C, and 15D illustrate different views of an examplemobile telephone with an auxiliary image capture device in a position toface a user and another position to face a scene together with a primaryimage capture device according to embodiments of the present disclosure;

FIG. 15E illustrates a cross-sectional view of another example mobiletelephone according to embodiments of the present disclosure; and

FIG. 16 illustrates perspective views of an example accessory with anauxiliary image capture device that can be attached to a mobiletelephone to face a scene together with a primary image capture deviceaccording to embodiments of the present disclosure.

DETAILED DESCRIPTION

The presently disclosed subject matter is described with specificity tomeet statutory requirements. However, the description itself is notintended to limit the scope of this patent. Rather, the inventors havecontemplated that the claimed subject matter might also be embodied inother ways, to include different steps or elements similar to the onesdescribed in this document, in conjunction with other present or futuretechnologies. Moreover, although the term “step” may be used herein toconnote different aspects of methods employed, the term should not beinterpreted as implying any particular order among or between varioussteps herein disclosed unless and except when the order of individualsteps is explicitly described.

In accordance with embodiments of the presently disclosed subjectmatter, primary and auxiliary image capture devices for image processingand related methods are disclosed herein. Each of the primary andauxiliary image capture devices includes an image sensor and a lens. Inan example use, the primary image capture device captures an image of ascene. The captured image may have a first quality characteristic, whichmay include a particular resolution, particular sensor size and/or pixelpitch, particular view angle and focal length, particular coloraccuracy, particular lens distortion characteristic, and the like. Theauxiliary image capture device captures another image of the scene. Thisother captured image may have a quality characteristic that is of lowerquality in at least one aspect than is the quality characteristic of theimage captured by the primary image capture device. One or moreparameters of one or both of the captured images may be adjusted tocreate multiple adjusted images for approximating or matching thequality characteristic of the image captured by the primary imagecapture device. For example, one or more of a cropping factor, size,focal length, and scaling of one or both of the images may be adjustedto match a resolution and/or approximate a field of view of the images.In another example, the distortion characteristics of at least one ofthe capture devices can be calculated and a transformative procedure canbe applied to the image data captured from that device to align it withthe image data from the other sensor. In another example, a color of oneimage can be adjusted to match or approximate a color of the otherimage. The adjusted images may be used for image processing. In anexample use, one or more stereoscopic still images or video sequencesmay be generated by performing a registration or a rectification processbased on the captured images and the adjusted images. In anotherexample, data obtained from the auxiliary and primary capture devices,such as, but not limited, to disparity map, can be used to recreate anew view using data from the main capture device using a depth imagebased rendering technique (DIBR) to create a stereoscopic pair ofimages.

In other embodiments of the presently disclosed subject matter, primaryand auxiliary capture devices may be used to calculate the proper stereobase when creating stereoscopic images by taking two picturesside-by-side. The disparity data calculated by both captured images canbe used to calculate the ideal stereo base to make a stereoscopic imagewith comfortable viewing attributes. Under the same embodiment, the livedata of the captured devices can be used to determine the properpositioning of the main capture device to take the second picture.

In other embodiment of the presently disclosed subject matter, thedisparity map calculated using the auxiliary and main images can bescaled up or down to create a different stereoscopic representationcompared to the one generated from the true separation of the twocaptured devices. This is particularly useful in cases where the idealstereo base is either larger or smaller to the stereo base provided bythe fixed separation of the two captured devices.

In other embodiment of the presently disclosed subject matter, thedisparity map calculated using the primary and auxiliary images may beused to identify objects with large or infinite disparity as a result ofeither moving objects or objects that are too close or too far away forcomfortable viewing. In such cases, objects with large disparity can bemoved from one image to another image, or can be moved within the sameimage, or can be removed from one image or both images to correct forsuch large disparity. Unfilled areas resulting from the movement orremoval of objects, can be filled using image data from the other imageor interpolated using data from the same image.

In other embodiments of the presently disclosed subject matter, primaryand auxiliary image capture devices may be used to generate still imageswithout blurring. In many situations, conventional cameras are limitedin their ability to capture high speed motion obtain while maintainingproper exposure. ISO gain can only increase to a certain amount withoutsignificant noise, and as such, the necessary shutter speed may beimpossible to achieve. This may result in image blurring at high motion.The presently disclosed subject matter, when implemented using a highspeed auxiliary camera system, provides a means of reducing and/orremoving this blur. For example, a primary image capture device may beused to capture an image of a scene. An auxiliary image capture devicemay be used to capture multiple other images of the scene. Further, amotion blur kernel may be determined based on the multiple other images.The motion blur kernel may then be applied to remove blur from the firstimage of the scene.

In other embodiments of the presently disclosed subject matter, primaryand auxiliary image capture devices may be used to generate a videosequence without handshaking. For example, a primary image capturedevice may be used to capture a sequence of images of a scene. Anauxiliary image capture device may be used to capture another sequenceof images of the scene. Next, a disparity map is generated. Further, anobject having the same disparity between two successive images in thesequences may be determined. In response to determining that the objecthas the same disparity, the object is identified as a non-moving object.Next, a shaking motion vector for the non-moving object is calculated.The shaking in a video and/or still image of the scene generated basedon the images may be compensated.

Stereoscopic, or three-dimensional, content is based on the principle ofhuman vision. Two separate detectors detect the same object—orobjects—in a plurality of images from slightly different angles andproject them onto two planes. The plurality of images is thentransferred to a processor which assigns the captured images as the oneview (i.e., left or right eye), analyzes the individual images, possiblyinterpolates additional frames/frame views, and for each image generatesa corresponding view for the other eye. Those two images or sequencesare then combined to create stereoscopic (three dimensional) content.The resulting three-dimensional content can further be encoded using,but not limited, to one of the video encoding popular formats such asJPEG, MPEG, H.264, etc. The generated content can further be stored withaudio to a digital media using one of popular containers such as .avi,.mpg, etc. In other embodiments of the presently disclosed subjectmatter, primary and auxiliary image capture devices may be used togenerate a focus stacking image. The main captured device can beconfigured to capture an image on the desired focal distance, whereasthe auxiliary sensor can be used to capture an image on a differentfocal distance. The data from the two captured devices can then becombined to create a focus stacking image.

Content captured using conventional image capture devices are oftenlimited in their ability to reproduce the dynamic range of a scene. Incertain scenes where very light and very dark objects coexist, it isimpossible for typical image sensors to obtain the correct exposureacross the image, and hence, impossible for the camera to reproduce whatthe user of the camera sees. The presently disclosed subject matterallows the two cameras in the system to properly adjust their exposuresettings to capture the bright and dark areas separately to create aHigh-Dynamic-Range (HDR) image of the scene with balanced bright anddark areas.

Embodiments of the presently disclosed subject matter may be implementedon a camera or other image capture system including primary andauxiliary image capture devices. Each of the primary and auxiliary imagecapture devices may include an image sensor and one or more lenses. Thecamera may be implemented on a mobile device such as a mobile phone orsmart phone. Further, embodiments of the presently disclosed subjectmatter may also be based on the technology that allows a user to capturea pair of two-dimensional video sequences, one from the primary imagecapture device of the camera and the other from the auxiliary imagecapture device for creating a three-dimensional video sequence. Thefunctions disclosed herein can be implemented in hardware, software,and/or firmware that can be executed within, for example, but notlimited to the proposed camera, a digital still camera, another type ofvideo camera (or camcorder), a personal computer, a digital pictureframe, a set-top box, an HDTV, a phone, or the like.

In an example, a portable capture device such as a cellphone, camera, ora tablet may include primary and auxiliary image capture devicesaccording to embodiments of the present disclosure. The auxiliary imagecapture device may be attached to or otherwise integrated with thecamera or cellphone for capturing one or more images of a scene. Data ofthe images captured by the auxiliary image capture device may be usedtogether with data of one or more images captured by the primary imagecapture device for creating a three-dimensional image or video sequence.In an example attachment of the auxiliary image capture device, theauxiliary image capture device can be placed on a back of a rotatableLCD display housing such that when the LCD display housing is in an openposition, the image plane of the auxiliary image capture device isparallel—or nearly parallel—to the image plane of the primary imagecapture device. On cellphones, the inwards facing camera that isprimarily used for video conferencing applications, can be used as theauxiliary capture device by modifying its design and allowing it toeither capture images from the person holding the cellphone or the scenewith the same field of view as the main camera. The auxiliary imagecapture device and its lens can have a quality characteristic that is oflower quality than a quality characteristic of the primary image capturedevice and its lens. It is noted that the quality and the resolution ofresulting three-dimensional image or video may depend directly on thequality and resolution of the primary image capture device and its lensand may not directly depend on the quality and resolution of theauxiliary image capture device and its lens. Such three-dimensionalstill pictures or video sequences can be viewed or displayed on asuitable stereoscopic display.

Method embodiments described herein can be implemented on a systemcapable of capturing still images or video sequences, displayingthree-dimensional images or videos, and executing computer executableinstructions on a processor. The device may be, for example, a digitalstill camera, a video camera (or camcorder), a personal computer, adigital picture frame, a set-top box, an HDTV, a phone, or the like. Thefunctions of the device may include methods for selecting videosegments, creating corresponding views for each image in the main videosequence, rectifying and registering at least two views, matching thecolor and edges of the views, performing stabilization of the sequence,calculating a sparse or dense depth map, synthesizing views, alteringthe perceived depth of objects, and any final display-specifictransformation to create a single, high-quality three-dimensional videosequence.

FIG. 1 illustrates a block diagram of an exemplary image capture system100 including a primary image capture device 102 and an auxiliary imagecapture device 104 for use in capturing images of a scene and performingimage processing according to embodiments of the presently disclosedsubject matter. In this example, the system 100 is a digital cameracapable of capturing multiple consecutive, still digital images of ascene. The devices 102 and 104 may each capture multiple consecutivestill digital image of the scene. In another example, the system 100 maybe a video camera capable of capturing a video sequence includingmultiple still images of a scene. In this example, the devices 102 and104 may each capture a video sequence including multiple still images ofthe scene. A user of the system 100 may position the system in differentpositions for capturing images of different perspective views of ascene. The captured images may be suitably stored and processed forgenerating three-dimensional images as described herein. For example,subsequent to capturing the images of the different perspective views ofthe scene, the system 100, alone or in combination with a computer suchas computer 106, may use the images for generating a three-dimensionalimage of the scene and for displaying the three-dimensional image to theuser.

Referring to FIG. 1, the primary and auxiliary image capture devices 102and 104 may include image sensors 108 and 110, respectively. The imagesensor 110 may be of a lesser quality than the image sensor 108. Forexample, the quality characteristics of images captured by use of theimage sensor 110 may be of lower quality than the qualitycharacteristics of images captured by use of the image sensor 108. Theimage sensors 108 and 110 may each include an array of charge coupleddevice (CCD) or CMOS sensors. The image sensors 108 and 110 may beexposed to a scene through lenses 112 and 114, respectively, and arespective exposure control mechanism. The lens 114 may be of lesserquality that the lens 112. The system 100 may also include analog anddigital circuitry such as, but not limited to, a memory 116 for storingprogram instruction sequences that control the system 100, together withat least one CPU 118, in accordance with embodiments of the presentlydisclosed subject matter. The CPU 118 executes the program instructionsequences so as to cause the system 100 to expose the image sensors 108and 110 to a scene and derive digital images corresponding to the scene.The digital image may be captured and stored in the memory 116. All or aportion of the memory 116 may be removable, so as to facilitate transferof the digital image to other devices such as the computer 106. Further,the system 100 may be provided with an input/output (I/O) interface 120so as to facilitate transfer of digital image even if the memory 116 isnot removable. The system 100 may also include a display 122controllable by the CPU 118 and operable to display the captured imagesin real-time for real-time viewing by a user.

The memory 116 and the CPU 118 may be operable together to implement animage processor 124 for performing image processing including generationof three-dimensional images in accordance with embodiments of thepresently disclosed subject matter. The image processor 124 may controlthe primary image capture device 102 and the auxiliary image capturedevice 104 for capturing images of a scene. Further, the image processor124 may further process the images and generate three-dimensional imagesas described herein. FIG. 2 illustrates a flow chart of an exemplarymethod for performing image processing using the system 100, alone ortogether with any other suitable device, in accordance with embodimentsof the present disclosure. Referring to FIG. 2, the method includesusing 200 a primary image capture device to capture a first image of ascene. For example, the image processor 124 shown in FIG. 1 may controlthe primary image capture device 102 to capture an image of a scene. Thecaptured image may have a particular quality characteristic, which mayinclude a particular resolution, particular sensor size and/or pixelpitch, particular view angle and focal length, particular coloraccuracy, particular lens distortion characteristic, and the like. Thecaptured image may be one of multiple images captured by the primaryimage capture device 102. The different images can include images of thesame or different perspective views of the scene. The CPU 106 may thenimplement instructions stored in the memory 104 for storing the capturedimages in the memory 104.

The method of FIG. 2 includes using 202 an auxiliary image capturedevice to capture another image of the scene. For example, the imageprocessor 124 shown in FIG. 1 may control the auxiliary image capturedevice 104 to capture an image of a scene. The image captured by theauxiliary image capture device 104 may have a particular qualitycharacteristic that is of lower quality than the quality characteristicof the image captured by the primary image capture device 102.

The method of FIG. 2 includes adjusting 204 one or more parameters ofone or both of the captured images to create a multiple, adjusted imagesfor approximating or matching a quality characteristic of the imagecaptured by the primary image capture device 102. For example, the imageprocessor 124 may adjust a cropping factor, size, scaling, the like, orcombinations thereof of one or both of the images captured by theprimary image capture device 102 and the auxiliary image capture device104 such that a resolution of the captured images are matched. Inanother example, the image processor 124 may adjusting a croppingfactor, size, scaling, the like, or combinations thereof of one or bothof the images captured by the primary image capture device 102 and theauxiliary image capture device 104 such that a field of view of othercaptured images is approximated. In another example, the image processor124 may adjust a color of one image to match or approximate a color ofthe other image.

The method of FIG. 2 includes utilizing 206 the adjusted images forimage processing. The image processing functions may include any ofregistration, rectification, disparity calculation, stereo-basecalculations, or depth-based rendering to generate stereoscopic imagesor video sequences, High-Dynamic-Range techniques to balance dark andbright areas or the scene, anti-blurring techniques to remove motionblurring, video stabilization techniques, and focus stacking functions.

FIGS. 3A and 3B illustrates perspective views of an example video camera300 including a primary image capture device and an auxiliary imagecapture device according to embodiments of the present disclosure. Thevideo camera 300 includes a rotatable component 302 that can bepositioned in a closed position as shown in FIG. 3A or an open positionas shown in FIG. 3B. A user can manually rotate the component 302 alonga hinge 304 for positioning the component 302 in either the open orclosed positions. In this example, the component 304 includes an LCDdisplay (not shown because it is located on an opposing side of thecomponent 304). In the open position, the LCD display can be viewed bythe user. On a facing side 306 of the component 304 (i.e., a sideopposing the LCD display), the auxiliary image capture device 308 ispositioned for capture of images in accordance with embodiments of thepresent disclosure.

In this example, a main body 310 of the video camera 300 includes theprimary image capture device. The primary image capture device 312 maybe positioned on a front portion of the main body 310. The captureddevices 308 and 312 may be positioned such that the image planes of theprimary and auxiliary image capture devices are parallel or nearlyparallel to one another. A micro-switch or a magnetic switch can beplaced between the rotatable portion 306 and the body 310 to inform theuser when the portion 306 has reached the proper position for capturing3D videos or other applications in accordance with embodiments of thepresent disclosure. This information can be displayed on the LCD screen,or by any other suitable technique such as turning on an indicator lightsuch as an LED.

FIG. 4 illustrates a perspective view of an example digital camera 400including a primary image capture device and an auxiliary image capturedevice according to embodiments of the present disclosure. Referring toFIG. 4, the camera 400 includes a main body 402 having a primary imagecapture device 404 and an auxiliary image capture device 406. Capturedevices 404 and 406 may be located at any suitable position on the body402.

In an embodiment, the resolution and the frame rate of the auxiliaryimage capture device may be the same as or equal to the resolution andthe frame rate of the primary image capture device. In situations inwhich the frame rates of the sensors are different, piece-wise constant,linear, or other suitable techniques of interpolation between frames canbe used to compensate for missing frames in a video sequence captured bythe auxiliary image capture device. This may apply to both cases inwhich output of the auxiliary image capture device is a video stream, ora set of consecutive image captures at constant or variable frame rates.

In addition, the quality and focal length of the lens of the auxiliaryimage capture device, and its zooming capabilities if applicable, may beequal to those of the lens of the primary image capture device. Thereare however limitations on the optics relationship of the main andauxiliary cameras. At the shortest focal length of the main cameras zoomlens, the auxiliary camera may have the same angle of view, hencenecessitating an equivalent focal length of the auxiliary camera equalto the widest focal length of the main zoom. Similarly, however, thereare practical limits to the amount of digital zoom that can beaccomplished while still being able to reasonably match data between thecameras, likely limiting the longest zoom focal length to somewherebetween 4× and 8× the shortest focal length, although this is not aspecific limitation of the design described herein. In cases in whichthe primary lens has zooming capabilities but the auxiliary lens doesnot, the image sensor of the auxiliary image capture device may havehigher resolution than the image sensor of the primary image capturedevice. This greater resolution may allow for more accurate “digitalzooming,” the process of which is discussed in further detail hereinbelow. It is noted that the optimal quality specifications for thesensors of the primary and auxiliary image capture devices should not beconsidered absolute requirements. In other words, the presentlydisclosed subject matter is not limited to the systems with optimalspecifications described herein above.

In accordance with embodiments of the present disclosure, the distancebetween the sensors of the primary and auxiliary image capture devicesand the angle between the two sensor planes can be constant or adaptive.In the latter case, the auxiliary sensor and lens may be attached to,for example, a sliding mechanism inside a rotatable portion, such as,the portion 302 shown in FIGS. 3A and 3B. The position and angle of theauxiliary sensor and lens can be, for example, controlled by a user,manually by a stepper-motor, or automatically based on computer controlthat determines a suitable distance between the auxiliary sensor and theprimary sensor depending on a scene being captured.

FIG. 5 illustrates steps in an example method for creating astereoscopic image pair in accordance with embodiments of the presentdisclosure. In an example embodiment, the stereoscopic image pair may becreated by modifying an image or video sequence captured using anauxiliary image capture device directly (as indicated by dashed lines inFIG. 5). This example technique may be best suited to high-qualityauxiliary image capture devices and is applicable to both video andstill applications.

In another example embodiment, a method may apply depth-based imagesynthesis processes on a plurality of image frames comprising a videosequence, as by the specific configuration of the hardware describedherein. While matching lens/sensor optics combinations for the primaryand auxiliary image capture devices may be utilized, in application,this is may be infeasible, owing to the zoom requirements of the maincamera and the space limitations for the auxiliary. In such a scenario,if video capture is required, this methodology is most suitable. In theexample of FIG. 5, the stereoscopic image pair may be created bymodifying an image or video sequence captured using an auxiliary imagecapture device indirectly using a depth map and view synthesis (asindicated by solid lines in FIG. 5).

Referring to FIG. 5, in a process using the auxiliary image capturedevice directly, a primary image capture device may capture a videosequence 500. In addition, the auxiliary image capture device maycapture a video sequence 502. The video sequences 500 and 502 may becaptured of the same scene at approximately the same time. The images ofthe video sequence 500 may be directly used without modification forcreating a three-dimensional video sequence 504 along with adjustedimages captured by the auxiliary image capture device. In a firstexample step of adjusting the images captured by the auxiliary imagecapture device, the images may be frame interpolated 506. In asubsequent, example step, a resolution of the images may be adjustedsuch as by, for example, digital zoom 508. In another subsequent,example step, a color and/or luminance of the images may be adjusted510. In direct 3D construction, the images may be suitable combined withthe images captured by the primary image capture device to result in thethree-dimensional video sequence 504.

In the example of creating the stereoscopic image pair by use of a depthmap and view synthesis (as indicated by solid lines in FIG. 5), a depthmap may be calculated by use of the images captured by the primary imagecapture device and the adjusted images of the auxiliary image capturedevice 512.

Finally, in accordance with another embodiment, possibly best suited tostill image capture and/or a low quality auxiliary camera system, theimage data from the auxiliary camera can be used to correlate to theprimary camera, extract minimum and maximum disparities to identify nearand far focus windows, and apply this data to calculate an optimalstereo baseline for the scene. The user can then be instructed in anynumber of ways to move to a proper offset for a second image capture tocomplete a stereo pair.

FIG. 6 illustrates a flow chart of an exemplary method for 3Dconstruction according to embodiments of the present disclosure. Thesteps of FIG. 6 can be decided during specification phase or may beimplemented by suitable hardware, software, and/or firmware. Forexample, the steps may be implemented by the image processor 124 shownin FIG. 1. Referring to FIG. 6, the method may be applied sequentiallyfor each frame from a main video sequence or image set (step 600). Themain video sequence or image set may be captured by a primary imagecapture device. Next, the method includes determining 602 whether themain and auxiliary images or frames are synched. It may be important tomake sure that each frame from the auxiliary sequence corresponds to oneand only one frame from the main sequence; therefore, if it is knownthat the frame rate for the auxiliary camera is different than the framerate of the main camera, a between-frame interpolation method can beused to achieve this objective. Since the optics of the two cameras arelikely incongruent, it is necessary to first configure their outputs sothat 3D registration, and hence depth extraction, can be performed for agiven image pair. Therefore, in response to determining that the mainand auxiliary images or frames are synched, the method may proceed tostep 604. Conversely, in response to determining that the main andauxiliary images or frames are not synched, the method may proceed tostep 606. It is noted that in case the synchronization informationbetween the two cameras is already available, such as absolute orrelative timestamps, or synchronization frames in both cameras alongwith known frame rates, the order in which the synchronization and imagemodification processes described below take place can be reversed. Thatis, first matching frames could be selected and then the viewmodification and color/luminance adjustments could be applied. Anexample of this technique is described in the example of FIG. 6.

At step 604, the method includes finding a matching image or frame fromthe auxiliary video sequence or image set. In an example, images orframes may be matched based on a time of capture. Other examples ofmatching images or frames are described in further detail herein.

Subsequent to step 604, the method may include adjusting 608 a size,resolution, and zoom level of the auxiliary frame to match thecorresponding main frame. For example, one or more of a cropping factor,size, and scaling of a captured image may be adjusted to match aresolution and approximate a field of view of a corresponding image.Other examples described in further detail herein.

Subsequent to step 608, the method may include adjusting 610 colorand/or luminance of the auxiliary frame to match the corresponding mainframe. For example, the color in an area of an auxiliary image may beadjusted to match the color in a corresponding area of the correspondingprimary image. In another example, regions of an overlapping field ofview of the image capture devices may be identified. In this example,the color properties of the regions may be extracted. Next, in thisexample, color matching and correction operations may be performed toequalize the images. An exemplary method for this purpose involvesleveling of each of the R, G, and B channels in the auxiliary frame suchthat the mean of the pixel colors equates to the mean of pixel colors inthe main frame. This method is more suitable if it is believed that theauxiliary frame can be directly used to create a stereoscopic pair ofimages along with the main frame. Alternatively, RGB values of allpixels in both frames could be adjusted so that their corresponding meanvalues would be neutral grey ([128,128,128]). This is not necessarilythe appropriate color “correction” for either camera, but rather, ameans of equating the camera outputs to make feature matching moreaccurate. This may be done globally, or within local windows across theimages to increase the accuracy of the correction. Other examplesdescribed in further detail herein.

Now referring to step 606, the method may include adjusting 612 colorand/or luminance of a number N frames of the auxiliary sequence to matchframes in the vicinity of a main frame. An example of this step isdescribed in further detail herein. Subsequently, the method may includeadjusting 614 color and/or luminance of the auxiliary frame to match thecorresponding main frame. Additionally, if this should be performedbefore attempting to synchronize the two cameras, the averaging isfurther extended to measure and correct for the mean over a windowednumber of frames, N. Subsequently, the method may include finding 614 amatching image or frame from the auxiliary video sequence or image set.Concurrent with these digital zoom and windowed color equalizationoperations, the actual process of synchronization may be performed. Thesame N windowed data used for color equalization can be used torecognize pixel patterns and movements within the frames from theindividual cameras, particularly tailored to the likely overlap regionof the cameras, the specifics of which may be dictated by the cameraoptics and the known separation (stereo base) of the cameras asphysically defined by the camera design. If any image stabilizationmechanism is used for the main camera, ideally either the informationabout such mechanism should be available at this time to be used on theauxiliary video sequence, or the same mechanism must be used on theauxiliary camera as well with precise synchronization between the twocameras.

The method of FIG. 6 may include determining 616 whether the auxiliaryframe is acceptable to be used directly. Criteria of acceptability fordirect use may include the comparative effective resolution of the twosensors (which will depend both on the absolute resolution, and thescaling change necessary due to any difference in angle of view betweenthe optics/camera combinations of the primary and auxiliary camera), theamount and degree of uncorrectable color differential between the twoimages, any difference in the noise profiles of the two cameras, and theamount and degree of uncorrectable distortion, as examples. It is notedthat a determination of whether to use the auxiliary image data can bemade at the time of system specification or design, or during operationof the system. For example, a determination can take place during systemspecification by examining the design characteristics of the twosensors. A decision on whether the auxiliary data can be used as is canbe determined by, for example, the relative resolution of the twosensors, their noise sensitivity, the characteristics of the two lenses,the desired quality, and the like. Also, experiments can be conducted ina laboratory setting to evaluate the relative performance of the twocameras and based on the outcome of such experiments to determinewhether the auxiliary image data can be used directly. In addition, suchdetermination can take place at the system level where a method thatexamines the quality characteristics of the two sensors for thisparticular scene can be implemented utilizing hardware and/or softwareresources of a combination of them to determine whether to use theauxiliary data directly. Such determination can also take place only onan area of the image and not the entire image. Various factors such asluminosity on an area, luminosity coupled with sensor sensitivity in anarea, lens distortion in an area, the like, and combinations thereof canbe used to form such determination, In response to determining that theauxiliary frame is acceptable, the method may proceed to step 618 wherethe auxiliary frame is used as the complementary frame.

Conversely, in response to determining that the auxiliary frame isunacceptable, the method may proceed to step 620 where a disparity mapis created. For example, a disparity map may be generated for ahorizontal positional offset of pixels of one or more images, such asstereoscopic still images. The method may include creating 622 a depthmap. For example, the disparity map may be used to generate the depthmap for a scene as described in further detail herein. The complementaryview may then be synthesized 624 for the main frame. It is noted thatthe final image can include blocks processed using paths 618 and otherblocks using paths 620, followed by 622, and 624. For example, in a lowlight scene the auxiliary sensor may have lower sensitivity compared tothe main sensor. In this case, the part of image that is dark can begenerated using image data from the main sensor utilizing thedepth-based rendering approach and the bright areas of the image may begenerated using image data obtained directly from the auxiliary sensor.

Subsequent to steps 618 and 624, the method may include displayingand/or saving 626 the 3D frame, which may include the main frame and theadjusted auxiliary frame. At step 628, the method includes determiningwhether there are additional frames or images to process. In response todetermining that there are additional frames or images, the method mayproceed to step 600 for processing the next frames. Otherwise, themethod may end at step 630.

FIG. 7 illustrates a flow chart of an exemplary method for framematching in accordance with embodiments of the present disclosure. Thismethod may be an example of steps 604 or 614 of the method of FIG. 6.The steps of FIG. 7 may be implemented by suitable hardware, software,and/or firmware. For example, the steps may be implemented by the imageprocessor 124 shown in FIG. 1. Referring to FIG. 7, the method may beginat step 700.

Subsequently, the method of FIG. 7 may include determining 700 whetheran exact matching frame can be found. For example, it may be determinedwhether an auxiliary frame matching a main frame can be found. Forexample, the auxiliary frame and main frame may have the same timestamp.In response to determining that the exact matching frame can be found,the matching frame may be selected from the auxiliary sequence (step704), and the method ends (step 706).

In response to determining that the exact matching frame cannot befound, the process may proceed to step 708. At step 708, the methodincludes determining whether timestamps are available for both frames.In response to determining that timestamps are available for bothframes, the method may include using between frame interpolation tosynthesize a matching frame (step 710). When timestamps are available,there is a known offset between auxiliary and main captured devices. Aweighted average technique based on the offset of the main and auxiliarycaptured devices can be used to synthesize the desired frame using imagedate from the previous and next frame in the target video sequence. Moreadvanced techniques can use motion estimation techniques to estimatemotion of moving objects and compensate for this factor as well.

In response to determining that timestamps are unavailable for bothframes, the method includes determining 712 whether a scene in the videosequence can be used to synchronize frames. This step involves framematching techniques where features are extracted from both main andauxiliary capture devices. The features of one frame in one sequence arecompared with the features in a number of frames on the other sequencethat are in the vicinity of the first sequence. In response todetermining that the scene in the video sequence cannot be used tosynchronize frames, it is determined that exact 3D construction is notpossible (step 714) and the method ends (step 706).

In response to determining that the scene in the video sequence can beused to synchronize frames, the method may include adjusting 716 a colorand/or view of the images, synchronizing 718 two sequences, and creating720 timestamps indicating the offset between frames in auxiliary andmain capture devices. The method may then proceed to step 702 forprocessing the next main frame.

FIG. 8 illustrates a flow chart of an exemplary method for image viewmatching according to embodiments of the present disclosure. This methodmay be an example of steps 606 or 608 of the method of FIG. 6. The stepsof FIG. 8 may be implemented by suitable hardware, software, and/orfirmware. For example, the steps may be implemented by the imageprocessor 124 shown in FIG. 1. Referring to FIG. 8, the method may beginat step 800.

Subsequently, the method includes determining 802 whether focal lengthinformation is available. In response to determining that focal lengthinformation is available, the method may include calculating 804 thematching view window on the auxiliary image. In response to determiningthat focal length information is unavailable, the method may includefinding 806 or creating the corresponding view on the auxiliary imageusing registration and perspective transformation. This may beaccomplished using a variety of methods including identification andmatching of key points, block matching on the boundaries, as well asother less sophisticated methods that look at various statistics on theimages, or combination of any other examples described herein.

The method of FIG. 8 also include cropping 808 the auxiliary image tomatch the field of view of the main image. The cropped auxiliary framemay be enlarged or reduced to match the size of the main frame (step810). Subsequently, the method may end (step 812).

Steps 806 and 808 can represent linear matrix operations on the imagedata. Combinations of these steps can be combined or separated in anynumber of ways without loss of generality. In a particularly fastembodiment for real-time processing, these steps, as well as alinearized approximation of distortion correction may be combined into asingle operation that can be implemented in a number of different waysincluding, but not limited to, applying a combined transformation thatperforms scaling, registration, lens correction, and the like at thesame time or a generalized look-up-table operation. The combinedtransform, combining any of the transformations described above, can becalculated by multiplying the matrices of the individual transforms.Referring to FIG. 5 for example the Digital Zoom step 508 may becombined with the registration operation implied in “Direct 3DConstruction”. In addition, a lens correction transformation can becombined with any of the other transformations. In FIG. 6, steps 606 or608 can contain both a scaling transformation and lens correction andcan be combined with the registration operation implied in step 618.

In the absence of a simple linear approximation for distortioncorrection, a methodology for combining transformation steps into apoint-wise look-up-table (LUT) or equivalent is still available. Thisinvolves pre-calculating, on a point-wise basis, the combination ofinverse transform steps to achieve a final result. For example, supposethat point A is projected to position B by distortion correction, toposition C by scaling, and to position D by perspective projectivetransformation to for alignment. By pre-calculating this traversal, onemay build an LUT for each end point in the final image, wherein it wouldbe stored e.g., point D results from an application of some transform topoint A of the source.

Constraints on the system can also be used to minimize the memorystorage required for such an LUT. For example, in a system in whichscaling will always imply enlargement, it is known that a single pixeloffset (e.g., from N to N+1) destination position may never result frommore than one pixel offset in the source (e.g., from M to M+1). Similarconstraints on the degree of distortion expected and the degree ofprojective correction needed for lens alignment can allow one to createan LUT in which an initial destination position N is set to correspondto source position S, and all subsequent pixels on a given row of dataare limited to an x coordinate change of [0 . . . 1] and a y coordinatechange of [−1 . . . 1]. In such a case, each subsequent pixel offset inthe row can be coded with a single bit for the change in X position inthe source data, and 2 bits for the possible changes in Y position. Asan example, for an HD video stream, which would imply at least 12 bitsto address X and Y coordinate positions, this represents a compressionof the required LUT by a factor of 8X.

FIG. 9 illustrates a flow chart of an exemplary method for color andluminance matching according to embodiments of the present disclosure.This method may be an example of steps 610 or 612 of the method of FIG.6. The steps of FIG. 9 may be either decided during system specificationphase, or during system design phase, or may be implemented by suitablehardware, software, and/or firmware on the system itself. For example,the steps may be implemented by the image processor 124 shown in FIG. 1.Referring to FIG. 9, the method may begin at step 900.

At step 902, the method may include determining whether colorcalibration information is available by the primary or auxiliary imagecapture devices. In response to determining that such color calibrationinformation is available, the method may include matching 904 theauxiliary fame's color and/or luminance to the main frame based on thecalibration information. Subsequently, the method may end (step 906).

In response to determining that color calibration information isunavailable, the method may proceed to step 908 where it is determinedwhether the frames will be used for direct or indirect 3D construction.In response to determining that the frames will be used for direct 3Dconstruction, the method may proceed to step 910 as described hereinbelow. In response to determining that the frames will be used forindirect 3D construction, the method may proceed to step 912 wherein itis determined whether registration/feature detection is needed. Inresponse to determining that detection is needed, the method includesmatching 914 color or luminance information of both frames to areference. In response to determining that detection is not needed themethod may implement step 914 and step 910 wherein color information ofthe auxiliary frame is matched to the main frame. Subsequently, themethod may end (step 906).

At this point, the system should have available, from each camera, imagepairs that are believed with some high confidence to be colorequivalent, synchronized, and representing equivalent views. In anembodiment, using auxiliary data directly, captured pairs may then besent to still image or video compression for creation of stereo 3D.Otherwise, extraction of pixel disparities then follows, for the purposeof converting disparity to depth. Any number of processes for disparityextraction may be utilized. Disparity, and in turn depth measures (asdictated by the equation,

$\left. {{depth} = \frac{{baseline} - {{focal}\mspace{14mu}{length}}}{disparity}} \right)$

must be measured with the specifics of the primary camera optics inmind, which is to say, the focal length of that cameras lens and thepixel pitch of that cameras sensor. For another embodiment of stereocreation, depths are then compared to find windows of minimum andmaximum depth, the information of which can be fed to a process thatcalculates an appropriate stereo base and directs the user on how totake a second capture to create the desired stereo pair.

Implementing an embodiment of 3D creation, for a given image frame inthe video sequence, having the raster data and the approximated depthmap, a corresponding image view (or views) may be synthesized. Anysuitable technique for depth-based image view synthesis may be utilized,although the preferred embodiment herein is the conversion of depthinformation to a Z-axis measure in pixels, followed by angular rotationof the raster data about the Y-axis. In this embodiment, the depthprofile of the initial image pair of a video sequence should be analyzedto determine the positioning of the viewer screen plane, such that somepercentage of pixels may fall in front or behind, and such that therotation occurs about the central point (0,0,0) that represents thecenter of the 2D image at a depth equal to the chosen distance of thescreen plane. This methodology affords one the ability to choose theperceived depth of the frame pairs (using a larger rotation angle tocreate a sense of larger stereo baseline, or vice-versa), possibly withinput from the autofocus mechanism of the camera as to the approximatefocus distance of the objects in frame (short distances dictate atargeted smaller stereo base, and longer distances, the opposite). Oneimage may be synthesized, using the initial raster data as the other ofa pair, or alternatively, two images may be synthesized with oppositerotations to generate left and right images. It is noted that the screenplane and angular rotation chosen for the first image of a sequencecannot be altered after the fact.

Creation of the final 3D video sequence involves video encoding of theindividual frame sequences in a manner consistent with 3D videorepresentation. Any suitable technique may be utilized.

FIGS. 10A and 10B depict a flow chart of an exemplary method forgenerating a high-dynamic range image using primary and auxiliary imagecapture devices according to embodiments of the present disclosure. Thesteps of FIGS. 10A and 10B may be implemented by suitable hardware,software, and/or firmware. For example, the steps may be implemented bythe image processor 124 shown in FIG. 1. Referring to FIGS. 10A and 10B,the method may begin at step 1000 wherein a system having primary andauxiliary image capture devices according to embodiments of the presentdisclosure may enter a live view mode for displaying images to a user.

The method of FIGS. 10A and 10B includes analyzing 1002 to determinebright and dark areas of the scene. Further, the method may includedetermining 1004 light conditions of one or more focus objects orobjects of interest. Subsequently, the method may include setting 1006capture characteristics of the primary image capture device for optimalexposure of the focus object.

The method of FIGS. 10A and 10B also include determining 1008 theexposure condition of remaining objects in the scene. In response todetermining that the objects have the same exposure, the method may end(step 1010). In response to determining there is overexposure, themethod may include decreasing 1012 sensitivity of the auxiliary imagecapture device to bring overexposed areas to ideal exposure conditions.In response to determining there is underexposure, the method mayinclude increasing 1014 sensitivity of the auxiliary image capturedevice to bring overexposed areas to ideal exposure conditions.

Subsequently, the method includes capturing 1016 images by firingsimultaneously the primary and auxiliary image capture devices. Next,the method may include performing 1018 image view matching of the imagecaptured by the auxiliary image capture device. For example, the methoddescribed with respect to FIG. 8 may be implemented. Next, the methodmay include performing color and/or luminance matching of the imagecaptured by the auxiliary image capture device. For example, the methoddescribed with respect to FIG. 9 may be implemented. The method may theninclude composing 1022 a final image by performing suitabletransformations between the image captured by the primary image capturedevice and the calibrated and equalized data of the auxiliary imagecapture device to compensate for underexposed or overexposed areas inimages captured by the primary image capture device as well as tominimize the disparity between the pixels in both images. The method maythen end (step 1024).

System and method embodiments for blur correction are disclosed herein.In an example, image blurring may result at high motion. The auxiliaryimage capture device may be used to correct image blurring as follows:given a high resolution low speed main camera (MC) and a low resolutionhigh speed second camera (SC), image blurring can be reduced byestimation motion flow in SC and use this information to recover theblur kernel in MC. The blur kernel is then used to remove blurring fromthe MC images.

FIG. 11 illustrates a flow chart of an exemplary method for removingmotion blur using a system having primary and auxiliary image capturedevices according to embodiments of the present disclosure. The steps ofFIG. 11 may be implemented by suitable hardware, software, and/orfirmware. For example, the steps may be implemented by the imageprocessor 124 shown in FIG. 1. Referring to FIG. 11, the method maybegin at step 1100 wherein a system having primary and auxiliary imagecapture devices according to embodiments of the present disclosure mayenter a motion de-blurring mode.

The method of FIG. 11 includes capturing 1102 high-speed images by useof the auxiliary image capture device. Further, the method may includefocusing and capturing 1104 a primary image using a primary imagecapture device. The last N number of image captured by the auxiliaryimage capture device may be used to calculate the motion blur kernel(step 1106). Subsequently, the method may include applying 1108 themotion blur kernel extracted from an auxiliary imager to remove blur.The method may then end (step 1110).

In accordance with embodiments disclosed herein, video stabilization maybe provided. For example, FIG. 12 illustrates a flow chart of anexemplary method for video stabilization in a system having primary andauxiliary image capture devices according to embodiments of the presentdisclosure. The steps of FIG. 12 may be implemented by suitablehardware, software, and/or firmware. For example, the steps may beimplemented by the image processor 124 shown in FIG. 1. Referring toFIG. 12, the method may begin at step 1200 wherein a system havingprimary and auxiliary image capture devices according to embodiments ofthe present disclosure may enter a video stabilization mode. In thismode, the primary and auxiliary image capture devices may capturesequences of images of a scene (step 1202) and equalize and calibratethe images in the video sequences (step 1204).

The method of FIG. 12 includes assisting 1210 in video and/or stillimage stabilization by adding an element to the calculations whencompensating 1212 for camera shake. Typical image stabilizationalgorithms rely on the evaluation and analysis of temporal displacementbetween frames (of video or other camera capture data), and are subjectto error introduced to the calculations by objects moving in the scene.Utilizing the auxiliary camera, offset between the two camera views fora single temporal instance (frame) can be calculated as an additive“structural” component for this process. Changes in this “structural”offset between temporally correlated blocks/regions of the image isindicative of moving objects in these location that may then be excludedfrom overall stabilization calculations. The method may then end (step1214). Next, the method includes creating 1206 a disparity map. Themethod also includes determining 1208 that objects in the images withdifferent disparity between two successive frames are moving objects,whereas objects with the same disparity are non-moving objects and theonly movement in this case is due to shaking.

The method of FIG. 12 includes calculating 1210 shaking motion vectors.Further, the method includes compensating 1212 for shaking. Shakingmotion vectors can be obtained by performing motion compensationanalysis between the captured frames, removing motion vectors of movingobjects, and using the remaining motion vectors to calculate the shakingvectors. The method may then end (step 1214).

In accordance with embodiments of the present disclosure, the primaryand auxiliary image capture devices may be components of a mobiletelephone. The auxiliary image capture device may be configured to facein a first position towards a user and to face in a second positiontowards a scene to be image captured. The auxiliary image capture devicemay include a mechanism for directing a path of light from the scenetowards the auxiliary image capture device such that the primary imagecapture device and the auxiliary image capture device can capture imagesof the scene. FIGS. 13A and 13B illustrate perspective views of anexample mobile telephone 1300 with an auxiliary image capture device1302 in a position to face a user and another position to face a scenetogether with a primary image capture device 1304 according toembodiments of the present disclosure.

Referring to FIG. 13A, the auxiliary image capture device 1302 and itsmirror mechanism 1306 may be positioned within the telephone 1300. Themirror mechanism 1306 may be mechanically or electronically rotated orotherwise moved to a position for directing light passing through anopening (not shown) toward the device 1302 for capturing an image of auser, for example. In another position shown in FIG. 13B, the mirrormechanism 1306 may be mechanically rotated or otherwise moved to aposition for directing light passing through an opening 1308 toward thedevice 1302 for capturing an image of a scene together with the device1304.

FIG. 13C illustrates a cross-sectional view of another example mobiletelephone according to embodiments of the present disclosure. Referringto FIG. 13C, the mirror can be replaced with two fixed light paths 1320and 1322 that can be constructed using a fiber optic material. One path1320 collecting the light from the scene and directing in one position1324 and the other one 1322 collecting the light from the inwards fieldof view and directing it in a different location 1326. By then movingthe sensor 1330 using a slide 1332 or any other manual or electronicallycontrolled mechanism to either 1324 or 1326 positions, capturing of theproper images according to the function of the phone is accomplished.

FIGS. 14A, 14B, and 14C illustrate different views of an example mobiletelephone 1400 with an auxiliary image capture device 1402 in a positionto face a user and another position to face a scene together with aprimary image capture device 1404 according to embodiments of thepresent disclosure. In this example, the auxiliary image capture device1402 may be rotated along a vertical or a horizontal axis to a positionshown in FIG. 14A for capturing an image of a user. The device 1402 maybe rotated to a position shown in FIG. 14C for capturing an image of ascene together with the device 1404.

FIGS. 15A, 15B, 15C, and 15D illustrate different views of an examplemobile telephone 1500 with an auxiliary image capture device 1502 in aposition to face a user and another position to face a scene togetherwith a primary image capture device 1504 according to embodiments of thepresent disclosure. In this example, the device 1502 is detachable fromthe body of the telephone 1500 for positioning on a front side or a rearside of the telephone 1500. The device 1502 may be attached to a frontside of the telephone 1500 for capturing images of the user as shown inFIGS. 15B and 15C. The device 1502 may be attached to a rear side of thetelephone 1500 for capturing images of a scene together with the device1504 as shown in FIGS. 15A and 15D.

FIG. 15E illustrates a cross-sectional view of another example mobiletelephone according to embodiments of the present disclosure. Referringto FIG. 15E, the inwards facing camera 1554, typically found in today'smodem cellphones 1550, can be as the auxiliary capture device. Thiscapture device 1554 resides on the opposite side of the primary capturedevice 1552 and a light guide 1558 that can be implemented as anattachable module 1556, can be used to direct light from the side facingthe primary camera 1560 to the inwards camera. When the module 1556 ispresent (FIG. 15E-C), the cellphone operates according to theembodiments of the present disclosure. When the module 1556 is notpresent, the cellphone resumes its typical operation where the inwardsfacing camera is used to take pictures or video of the person holdingthe cellphone.

FIG. 16 illustrates perspective views of an attachable, computing deviceaccessory 1600 including an auxiliary image capture device 1602 that isconfigured to be attached to a cellphone or other computing device 1601to face a scene together with a primary image capture device 1603according to embodiments of the present disclosure. A typical cellphoneor other computing device may have a connector 1604 that allows thephone to receive power as well as to communicate with external devices.In addition, other ports may exist on the side of the phone with theconnector. Such ports may be buttons, ports to connect the phone toexternal speakers or earphones 1606 and the like. Also, there may beopenings for speakers, microphones 1608 or any other interface thatallows the phone to communicate, or transmit, or receive any signal thatis coming from the outside world. The accessory 1600 may have a casingthat allows the accessory to attach to the cellphone. The accessory 1600may have a replicated set of ports, connectors, or openings that matchthe ones of the cellphone. Each set includes the port that attaches tothe phone and the one that attaches to the outside world. For examplethe connector 1615 of the accessory 1600 attaches to the connector 1604of the cellphone. Also, the connector 1614 of the accessory 1600 can bephysically connected with the connector 1615 of the accessory 1600 toallow other devices to communicate with the cellphone when the auxiliarycapture device is not used. Similarly, port 1617 of accessory 1600 isconnected to port 1606 of the cellphone, whereas port 1616 is physicallyconnected with port 1617 of the accessory 1600. The openings formed by1618 and 1619 can also allow signals generated or received on 1608 topass to the outside world.

The auxiliary capture device 1602 receives power from the 1604 connectorwith which is attached utilizing a digital or analog cable 1622. Imagedate pass through the cable 1622 as well. It may also be required tohave one or more integrated circuits 1624 and/or other passiveelectronic components to translate the signals from capture device tosomething that the cellphone understands.

The auxiliary capture device can be also on track so it can slide on thehorizontal axis to change the stereo base when creating a stereoscopicimage.

Although FIG. 16 shows a position of the auxiliary capture devicesuitable for processing when phone is oriented in landscape mode, itshould be also noted that the auxiliary sensor can reside anywhere inthe phone and can be placed in a way that processing can be implementedin portrait mode (position 1640). Also, it is possible to include twoauxiliary sensors (one in position 1602 and another one in position1640) to enable processing in both landscape and portrait orientations.

The various techniques described herein may be implemented with hardwareor software or, where appropriate, with a combination of both. Thus, themethods and apparatus of the disclosed embodiments, or certain aspectsor portions thereof, may take the form of program code (i.e.,instructions) embodied in tangible media, such as floppy diskettes,CD-ROMs, hard drives, or any other machine-readable storage medium,wherein, when the program code is loaded into and executed by a machine,such as a computer, the machine becomes an apparatus for practicing thepresently disclosed subject matter. In the case of program codeexecution on programmable computers, the computer will generally includea processor, a storage medium readable by the processor (includingvolatile and non-volatile memory and/or storage elements), at least oneinput device and at least one output device. One or more programs arepreferably implemented in a high level procedural or object orientedprogramming language to communicate with a computer system. However, theprogram(s) can be implemented in assembly or machine language, ifdesired. In any case, the language may be a compiled or interpretedlanguage, and combined with hardware implementations.

The described methods and apparatus may also be embodied in the form ofprogram code that is transmitted over some transmission medium, such asover electrical wiring or cabling, through fiber optics, or via anyother form of transmission, wherein, when the program code is receivedand loaded into and executed by a machine, such as an EPROM, a gatearray, a programmable logic device (PLD), a client computer, a videorecorder or the like, the machine becomes an apparatus for practicingthe presently disclosed subject matter. When implemented on ageneral-purpose processor, the program code combines with the processorto provide a unique apparatus that operates to perform the processing ofthe presently disclosed subject matter.

While the embodiments have been described in connection with thepreferred embodiments of the various figures, it is to be understoodthat other similar embodiments may be used or modifications andadditions may be made to the described embodiment for performing thesame function without deviating therefrom. Therefore, the disclosedembodiments should not be limited to any single embodiment, but rathershould be construed in breadth and scope in accordance with the appendedclaims.

1-13. (canceled)
 14. An apparatus for creating an image of a scene, theapparatus comprising: at least two image capture devices comprising ofan image sensor and other components assisting in the capture process,wherein; the first image capture device comprising a first image-qualitycharacteristic; the second image capture device comprising a secondimage-quality characteristic; a processor, wherein said processor isconfigured to: receive video data from the first and the second imagecapture devices; for a pair of corresponding images obtained from thevideo data from the first and second capture devices; adjust the imageobtained from one image capture device to match at least one of thecharacteristics of the image obtained from the other image capturedevice, thus forming a modified pair of images; calculate the disparitybetween the modified pair of images; determine the areas of the imageswith constant disparity; calculate the motion vector of the areas withconstant disparity between frames; utilize the motion vector to performvideo stabilization in one of the received videos;
 2. The apparatus ofclaim 1, wherein processor adjusts one of a cropping factor, size, andscaling of one of the images of the image pair.
 3. The apparatus ofclaim 1, wherein processor uses focal lengths and optical properties ofthe primary and auxiliary image capture devices to calculate anoverlapping field of view and resolution, crop, and scale changes. 4.The apparatus of claim 1, wherein processor identifies distortioncharacteristics of the image capture devices; and performs atransformative procedure to correct distortion for equalizing thecaptured image pair.
 5. The apparatus of claim 1, wherein processoradjusts a color of one image of the captured image pair to one of matchand approximate a color of the other image.
 6. The apparatus of claim 1,wherein processor performs motion compensation to identify the movingfrom the non-moving areas of the scene.
 7. The apparatus of claim 1,wherein processor extracts image features using one or more pixels fromthe images in the captured image pair; matches extracted image featuresbetween the images in the captured image pair; computes a transformationto adjust at least one of the matched image features between the imagesin the captured image pair; applies the transformation to at least oneof the images in the captured image pair creating an adjusted image; andforms a transformed image pair by selecting two images from a secondgroup of images including the adjusted and captured images.
 10. Anapparatus for creating an image of a scene, the apparatus comprising: atleast two image capture devices comprising of an image sensor and othercomponents assisting in the capture process, wherein; the first imagecapture device comprising a first image-quality characteristic; thesecond image capture device comprising a second image-qualitycharacteristic; a processor, wherein said processor is configured to:receive still image data from the first image capture device; receivevideo data from the second image capture device; utilize a plurality ofvideo frames obtained from the second image capture device to calculatea motion blur kernel of the apparatus; and apply the motion blur kernelto the image obtained from the first image capture device to create anenhanced image.
 11. The apparatus of claim 10, wherein processorperforms motion compensation to identify the motion vectors.